The present invention relates generally to computer data compression and more particularly, to data deduplication processing to replace redundant data with space saving referential pointers.
Data deduplication is a data compression technique to eliminate duplicate copies of repeating data. Data deduplication can be used to improve storage utilization and can also be applied to network data transfers to reduce the number of bytes that must be sent in the data transfer. In the deduplication process, unique chunks of data and/or byte patterns, are identified and redundant chunks of data are replaced with a reference pointer to link toward a respective and corresponding unique chunk of data. When data deduplication is used in backups and data transfers, a unique index can be maintained to map link references and store a record of the data chunk size used to perform a deduplication.